Univariate Relationships with Absolute Deviation
MPT Reanalysis Project (code and text current document: Henrik Singmann)
Results from: 2022-07-01
- Overview
- DV: Absolute Deviation from Complete Pooling MLE Estimate
- DV: Absolute Deviation from Latent Trait Partial Pooling Estimate
Overview
This document provides an overview of the univariate relationships of all covariates with the absolute parameter deviation. We separate the relationships by focussing on one method as the target method and then investigating the relationships for each of the remaining methods with this method.
DV: Absolute Deviation from Complete Pooling MLE Estimate
We begin by investigating the absolute relationship from the simplest
method, the complete pooling MLE method (i.e., y always
refers to Comp MLE" and x refers to the other
method in the pair). This leaves us with 10623 observations for the
analysis.
Effect of Method
We can also look at the histogram of the absolute deviation across methods.
Effects of Continuous Covariates
In the following plots, the blue line shows the fitted model (in case it is not a simple linear relationship, the transformation of the independent variable is given in parentheses in the x-axis label), The RMSE value shown in the plot is the RMSE of this model (i.e., the blue line). The red line shows a GAM on the independent variable with shrinkage applied thin plate regression spline and the red number is the corresponding RMSE.
In case observations had to be removed for the analysis, the
percentage of removed (rem) observations is also shown in
the x-axis caption. In some cases, some observations are filtered to
show a clearer picture, this is indicated with filter.
Standard Error
Rho (only pairs on same branch)
Effect of Parameter Estimate
Hetereogeneity
The data suggests a step-like relationship such that only values that
are at or near zero show a considerable probability of non-zero absolute
deviations. To look at this further, we can see how probable it is to
observe values near zero. The following table shows that at least 80% of
observations have a log1p value that is very near to
zero.
## # A tibble: 8 x 3
## cond_x less_than_00001 less_than_01
## <fct> <dbl> <dbl>
## 1 Comp Bayes 0.791 0.886
## 2 No asy 0.791 0.886
## 3 No PB 0.791 0.885
## 4 No NPB 0.807 0.902
## 5 No Bayes 0.781 0.880
## 6 Beta PP 0.789 0.884
## 7 Trait_u PP 0.852 0.894
## 8 Trait PP 0.791 0.886We can also get this table for the p-value itself (i.e., not
log1p) which is identical as for very small values
log1p is approximately equal to p.
## # A tibble: 8 x 3
## cond_x less_than_00001 less_than_01
## <fct> <dbl> <dbl>
## 1 Comp Bayes 0.791 0.886
## 2 No asy 0.791 0.886
## 3 No PB 0.791 0.885
## 4 No NPB 0.807 0.902
## 5 No Bayes 0.781 0.880
## 6 Beta PP 0.789 0.884
## 7 Trait_u PP 0.852 0.894
## 8 Trait PP 0.791 0.886If we look at the conditional distribution of absolute deviation whether or not it is very near to zero, we can see that there is some evidence for the step-like relationship, but the pattern is not overwhelming.
Rho
Fungibility
Fungibility (only pairs on same branch)
Model Fit
Proportion of Individual Misfit
Relative Parameter Information
The reason both plots look pretty much the same is that both relative
parameter information variables are highly correlated, \(r \approx 1\). We therefore focus on one of
the two below (rel_par_weight_y).
Using a logarithm:
And removing all with a relative weight of roughly 1:
Relative Parameter N
We can also consider the relative N. As tehse are again highly
correlated (\(r \approx 1\)), we use
y again exclusivly:
Here it makes sense to trim the x-axis a bit:
Effects of Categroical Covariates
The effect of covariate is shown in two ways. The table below all plots gives the RMSEs for the model given the covariate across all comparison methods. In case the number of levels is not too large, a plot of the difference in absolute deviation conditional on the factor levels is shown. Some factor levels may be removed for plotting (i.e., those levels for which the proportion of observations is less than 0.04). In this case, the number of removed levels is also given.
Meta Data
## # A tibble: 4 x 10
## covariate nlevels `No asy` `No PB` `No NPB` `No Bayes` `Comp Bayes` `Trait PP` `Trait_u PP` `Beta PP`
## <chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 model 8 0.0598 0.0530 0.0563 0.0725 0.0617 0.0599 0.0747 0.0747
## 2 model2 12 0.115 0.0958 0.108 0.115 0.0645 0.0907 0.188 0.165
## 3 parameter 49 0.485 0.429 0.418 0.413 0.0962 0.362 0.458 0.424
## 4 dataset 147 0.202 0.194 0.213 0.327 0.823 0.191 0.296 0.349Given that there are more than eight levels of model2,
we also look at a table of the mean absolute deviations:
## # A tibble: 12 x 10
## model2 prop `Comp Bayes` `No asy` `No PB` `No NPB` `No Bayes` `Beta PP` `Trait_u PP` `Trait PP`
## <fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2htsm_4 0.301 0.00590 0.0288 0.0362 0.0335 0.0490 0.0132 0.0121 0.0199
## 2 2htsm_5d 0.0890 0.0179 0.0583 0.0803 0.0682 0.109 0.0313 0.0360 0.0420
## 3 2htsm_6e 0.0226 0.00946 0.0574 0.0746 0.0666 0.0947 0.0190 0.0273 0.0415
## 4 c2ht6 0.0271 0.00371 0.0229 0.0208 0.0208 0.0253 0.0587 0.0529 0.0331
## 5 c2ht8 0.00565 0.000697 0.0118 0.0107 0.0110 NA 0.0178 0.0647 0.0263
## 6 pd_s 0.0316 0.00123 0.0167 0.0189 0.0189 0.0346 0.0230 0.00839 0.0114
## 7 pd_e 0.00753 0.0260 0.126 0.121 0.122 0.146 0.142 0.118 0.0462
## 8 pm 0.105 0.0610 0.0260 0.0296 0.0288 0.0966 0.0158 0.0343 0.0369
## 9 hb 0.0524 0.0316 0.0432 0.0506 0.0406 0.0855 0.0355 0.0191 0.0518
## 10 rm 0.149 0.000335 0.0110 0.0109 0.0109 0.0147 0.0106 0.0148 0.0142
## 11 real 0.104 0.00632 0.0208 0.0270 0.0225 0.0469 0.0137 0.0151 0.0173
## 12 quad 0.105 0.00762 0.0537 0.0658 0.0538 0.0746 0.0220 0.0278 0.0336We can also look at a table of the mean absolute deviations as a function of the parameter:
## # A tibble: 49 x 10
## parameter prop `Comp Bayes` `No asy` `No PB` `No NPB` `No Bayes` `Beta PP` `Trait_u PP` `Trait PP`
## <fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2htsm_4:b 0.0752 0.00452 0.0297 0.0479 0.0466 0.0379 0.0112 0.0121 0.0187
## 2 2htsm_4:d 0.0752 0.0126 0.0504 0.0584 0.0498 0.103 0.0219 0.0206 0.0332
## 3 2htsm_4:D 0.0752 0.00393 0.00784 0.00668 0.00668 0.0252 0.00836 0.00799 0.0101
## 4 2htsm_4:g 0.0752 0.00253 0.0273 0.0318 0.0307 0.0300 0.0113 0.00785 0.0175
## 5 2htsm_5d:b 0.0178 0.00130 0.0241 0.0384 0.0376 0.0453 0.00771 0.0275 0.0327
## 6 2htsm_5d:D 0.0178 0.00132 0.00698 0.00707 0.00707 0.0253 0.00328 0.00919 0.0104
## 7 2htsm_5d:d_1 0.0178 0.0373 0.107 0.137 0.115 0.213 0.0642 0.0582 0.0674
## 8 2htsm_5d:d_2 0.0178 0.0370 0.105 0.166 0.130 0.195 0.0585 0.0619 0.0733
## 9 2htsm_5d:g 0.0178 0.0126 0.0482 0.0531 0.0513 0.0673 0.0229 0.0229 0.0261
## 10 2htsm_6e:b 0.00377 0.000864 0.0202 0.0385 0.0363 0.0519 0.00890 0.0350 0.0481
## 11 2htsm_6e:d_1 0.00377 0.0266 0.119 0.185 0.165 0.204 0.0470 0.0333 0.0469
## 12 2htsm_6e:D_1 0.00377 0.000697 0.00727 0.00964 0.00906 0.0476 0.00183 0.0203 0.0197
## 13 2htsm_6e:d_2 0.00377 0.0192 0.145 0.152 0.130 0.128 0.0357 0.0371 0.0832
## 14 2htsm_6e:D_2 0.00377 0.000454 0.0106 0.0109 0.00822 0.0528 0.00270 0.0230 0.0193
## 15 2htsm_6e:g 0.00377 0.00888 0.0418 0.0522 0.0515 0.0834 0.0181 0.0152 0.0321
## 16 c2ht6:Dn 0.00904 0.00754 0.0266 0.0258 0.0258 0.0382 0.106 0.0980 0.0580
## 17 c2ht6:Do 0.00904 0.00104 0.0206 0.0204 0.0204 0.0204 0.0206 0.0222 0.0171
## 18 c2ht6:g 0.00904 0.00256 0.0217 0.0163 0.0163 0.0174 0.0490 0.0383 0.0244
## 19 c2ht8:Dn 0.00188 0.00149 0.0170 0.0147 0.0157 NA 0.0321 0.136 0.0507
## 20 c2ht8:Do 0.00188 0.000120 0.00850 0.00976 0.00976 NA 0.00536 0.0151 0.0117
## 21 c2ht8:g 0.00188 0.000483 0.00986 0.00762 0.00762 NA 0.0158 0.0426 0.0166
## 22 hb:b 0.0131 0.0586 0.113 0.130 0.0909 0.0774 0.0639 0.0284 0.0968
## 23 hb:c 0.0131 0.0631 0.0359 0.0470 0.0294 0.214 0.0667 0.0277 0.0906
## 24 hb:rc 0.0132 0.00244 0.0122 0.0134 0.0233 0.0306 0.00625 0.00657 0.00813
## 25 hb:re 0.0131 0.00207 0.0115 0.0142 0.0190 0.0199 0.00511 0.0137 0.0117
## 26 pd_e:A_alt 0.00452 0.0158 0.0831 0.0859 0.0859 0.112 0.105 0.0987 0.0404
## 27 pd_e:C_Exclusion 0.00151 0.0102 0.0905 0.0892 0.0892 0.109 0.106 0.0482 0.0191
## 28 pd_e:C_Inclusion 0.00151 0.0724 0.291 0.258 0.265 0.283 0.288 0.247 0.0908
## 29 pd_s:A 0.0158 0.000798 0.0132 0.0167 0.0166 0.0148 0.00772 0.00541 0.00995
## 30 pd_s:C 0.0158 0.00166 0.0202 0.0211 0.0211 0.0544 0.0382 0.0114 0.0129
## 31 pm:C1 0.0264 0.0637 0.0191 0.0117 0.0114 0.0753 0.00871 0.0176 0.0169
## 32 pm:C2 0.0264 0.0742 0.0112 0.00304 0.00304 0.0819 0.00458 0.0183 0.0150
## 33 pm:M 0.0264 0.0808 0.0392 0.0773 0.0746 0.177 0.0253 0.0415 0.0607
## 34 pm:P 0.0264 0.0253 0.0344 0.0263 0.0263 0.0521 0.0247 0.0598 0.0550
## 35 quad:ACbb1 0.0210 0.000587 0.0208 0.0229 0.0229 0.0570 0.00743 0.0182 0.0172
## 36 quad:ACwg1 0.0210 0.000801 0.0184 0.0206 0.0206 0.0536 0.00579 0.0179 0.0178
## 37 quad:D1 0.0210 0.00102 0.0124 0.0123 0.0123 0.00745 0.00555 0.0205 0.0207
## 38 quad:G1 0.0210 0.000565 0.0279 0.0277 0.0278 0.0160 0.00513 0.00629 0.0138
## 39 quad:OB1 0.0210 0.0351 0.189 0.246 0.186 0.239 0.0861 0.0760 0.0987
## 40 real:A1 0.0147 0.00486 0.0196 0.0334 0.0224 0.0356 0.0101 0.0108 0.0106
## 41 real:A2 0.0147 0.00479 0.0136 0.0311 0.0189 0.0298 0.00898 0.00832 0.00867
## 42 real:L1 0.0147 0.00526 0.0167 0.0191 0.0176 0.0481 0.0142 0.0160 0.0161
## 43 real:L2 0.0147 0.00497 0.0162 0.0146 0.0151 0.0455 0.0127 0.0164 0.0163
## 44 real:L3 0.0152 0.00632 0.0204 0.0206 0.0204 0.0590 0.0151 0.00737 0.00791
## 45 real:L4 0.0152 0.00588 0.0198 0.0192 0.0197 0.0587 0.0148 0.00649 0.00734
## 46 real:Re 0.0147 0.0122 0.0396 0.0543 0.0435 0.0520 0.0199 0.0402 0.0544
## 47 rm:a 0.0497 0.000203 0.00298 0.00300 0.00300 0.00616 0.00314 0.00425 0.00448
## 48 rm:b 0.0497 0.000131 0.0110 0.0109 0.0109 0.0119 0.00754 0.00573 0.00650
## 49 rm:r 0.0497 0.000670 0.0189 0.0187 0.0187 0.0259 0.0212 0.0343 0.0317Or at the standard deviation of the absolute deviations as a function of the parameter:
## # A tibble: 49 x 10
## parameter prop `Comp Bayes` `No asy` `No PB` `No NPB` `No Bayes` `Beta PP` `Trait_u PP` `Trait PP`
## <fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2htsm_4:b 0.0752 0.00705 0.0344 0.0601 0.0583 0.0439 0.0120 0.0163 0.0227
## 2 2htsm_4:d 0.0752 0.0223 0.0571 0.0835 0.0557 0.0976 0.0283 0.0209 0.0354
## 3 2htsm_4:D 0.0752 0.00415 0.0105 0.00961 0.00961 0.0235 0.00774 0.00946 0.0109
## 4 2htsm_4:g 0.0752 0.00352 0.0288 0.0377 0.0410 0.0312 0.0129 0.0121 0.0288
## 5 2htsm_5d:b 0.0178 0.00133 0.0188 0.0275 0.0249 0.0325 0.00595 0.0168 0.0217
## 6 2htsm_5d:D 0.0178 0.00107 0.00585 0.00576 0.00576 0.0122 0.00295 0.00684 0.00726
## 7 2htsm_5d:d_1 0.0178 0.0362 0.0729 0.0812 0.0717 0.140 0.0468 0.0406 0.0453
## 8 2htsm_5d:d_2 0.0178 0.0394 0.0798 0.120 0.105 0.159 0.0503 0.0363 0.0505
## 9 2htsm_5d:g 0.0178 0.0115 0.0282 0.0276 0.0291 0.0325 0.0156 0.0224 0.0240
## 10 2htsm_6e:b 0.00377 0.000230 0.0165 0.0345 0.0348 0.0302 0.00664 0.0236 0.0282
## 11 2htsm_6e:d_1 0.00377 0.0317 0.0583 0.123 0.113 0.183 0.0459 0.0300 0.0577
## 12 2htsm_6e:D_1 0.00377 0.000203 0.00582 0.00551 0.00495 0.0200 0.00139 0.00760 0.00815
## 13 2htsm_6e:d_2 0.00377 0.0236 0.0251 0.0715 0.0815 0.150 0.0377 0.0293 0.0468
## 14 2htsm_6e:D_2 0.00377 0.000220 0.00931 0.00675 0.00513 0.0223 0.00211 0.0122 0.0117
## 15 2htsm_6e:g 0.00377 0.00838 0.0246 0.0235 0.0229 0.0459 0.0112 0.00836 0.0282
## 16 c2ht6:Dn 0.00904 0.00596 0.0226 0.0227 0.0227 0.0236 0.111 0.0601 0.0332
## 17 c2ht6:Do 0.00904 0.00150 0.0179 0.0165 0.0165 0.0150 0.0191 0.0140 0.00981
## 18 c2ht6:g 0.00904 0.00193 0.0153 0.0138 0.0138 0.00871 0.0501 0.0271 0.0141
## 19 c2ht8:Dn 0.00188 0.000770 0.0123 0.0114 0.0114 NA 0.0247 0.0833 0.0230
## 20 c2ht8:Do 0.00188 0.0000935 0.00537 0.00430 0.00430 NA 0.00326 0.0218 0.00966
## 21 c2ht8:g 0.00188 0.000271 0.00326 0.00198 0.00198 NA 0.00536 0.0307 0.0114
## 22 hb:b 0.0131 0.0647 0.0860 0.0962 0.0572 0.0614 0.0603 0.0145 0.0567
## 23 hb:c 0.0131 0.0777 0.0480 0.0459 0.0260 0.0787 0.0672 0.0321 0.0954
## 24 hb:rc 0.0132 0.00307 0.0366 0.0365 0.0521 0.0299 0.00570 0.00304 0.00746
## 25 hb:re 0.0131 0.00285 0.0287 0.0300 0.0410 0.0221 0.00546 0.00827 0.0115
## 26 pd_e:A_alt 0.00452 0.0177 0.0602 0.0587 0.0587 0.102 0.0291 0.0394 0.0205
## 27 pd_e:C_Exclusion 0.00151 0.0113 0.0708 0.0726 0.0726 0.0723 0.0532 0.00543 0.0175
## 28 pd_e:C_Inclusion 0.00151 0.0851 0.0176 0.0290 0.0187 0.263 0.0654 0.00865 0.0490
## 29 pd_s:A 0.0158 0.000617 0.00979 0.0128 0.0129 0.0148 0.0146 0.00533 0.00887
## 30 pd_s:C 0.0158 0.00222 0.0168 0.0186 0.0186 0.0392 0.0751 0.00937 0.0110
## 31 pm:C1 0.0264 0.207 0.0204 0.0202 0.0202 0.142 0.0137 0.0129 0.0167
## 32 pm:C2 0.0264 0.240 0.0156 0.00715 0.00715 0.148 0.00325 0.0156 0.0138
## 33 pm:M 0.0264 0.239 0.0466 0.0800 0.0800 0.152 0.0277 0.0532 0.0791
## 34 pm:P 0.0264 0.0836 0.0359 0.0334 0.0345 0.0564 0.0322 0.0440 0.0452
## 35 quad:ACbb1 0.0210 0.000956 0.0117 0.0112 0.0112 0.0185 0.00530 0.0146 0.0122
## 36 quad:ACwg1 0.0210 0.00129 0.0115 0.0113 0.0113 0.0179 0.00586 0.0119 0.0106
## 37 quad:D1 0.0210 0.000965 0.00613 0.00628 0.00628 0.00594 0.00450 0.0131 0.0144
## 38 quad:G1 0.0210 0.000653 0.0203 0.0186 0.0192 0.0141 0.00425 0.00659 0.00913
## 39 quad:OB1 0.0210 0.0518 0.112 0.186 0.120 0.155 0.0789 0.0738 0.0880
## 40 real:A1 0.0147 0.00549 0.0212 0.0661 0.0333 0.0392 0.00836 0.0110 0.0118
## 41 real:A2 0.0147 0.00587 0.00916 0.0628 0.0268 0.0189 0.00637 0.00955 0.00560
## 42 real:L1 0.0147 0.00339 0.0116 0.0124 0.0119 0.0140 0.00722 0.0142 0.0145
## 43 real:L2 0.0147 0.00302 0.00812 0.00816 0.00710 0.0115 0.00545 0.0142 0.0129
## 44 real:L3 0.0152 0.00402 0.00812 0.00851 0.00812 0.0101 0.00581 0.00567 0.00645
## 45 real:L4 0.0152 0.00360 0.00836 0.00836 0.00813 0.00996 0.00433 0.00454 0.00596
## 46 real:Re 0.0147 0.0132 0.0599 0.149 0.112 0.0538 0.0128 0.0287 0.0374
## 47 rm:a 0.0497 0.000229 0.00272 0.00272 0.00272 0.00501 0.00379 0.00524 0.00564
## 48 rm:b 0.0497 0.000124 0.0107 0.0109 0.0108 0.0106 0.00713 0.00683 0.00732
## 49 rm:r 0.0497 0.000789 0.0215 0.0216 0.0216 0.0260 0.0264 0.0377 0.0227Categorical Covariates
## # A tibble: 2 x 10
## covariate nlevels `No asy` `No PB` `No NPB` `No Bayes` `Comp Bayes` `Trait PP` `Trait_u PP` `Beta PP`
## <chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 population 5 0.0170 0.00710 0.00712 0.00353 0.00206 0.0273 0.00558 0.0210
## 2 sci_goal 2 0.00288 0.00419 0.00405 0.00638 0.00101 0.000177 0.0287 0.0197DV: Absolute Deviation from Latent Trait Partial Pooling Estimate
In the second analysis, we focus on investigating the absolute
deviation from the most complex method, the latent trait partial pooling
method (i.e., y always refers to Trait PP and
x refers to the other method in the pair). This leaves us
with 10623 observations for the analysis.
Effect of Method
We can also look at the histogram of the absolute deviation across methods.
Effects of Continuous Covariates
In the following plots, the blue line shows the fitted model (in case it is not a simple linear relationship, the transformation of the independent variable is given in parentheses in the x-axis label), The RMSE value shown in the plot is the RMSE of this model (i.e., the blue line). The red line shows a GAM on the independent variable with shrinkage applied thin plate regression spline and the red number is the corresponding RMSE.
In case observations had to be removed for the analysis, the
percentage of removed (rem) observations is also shown in
the x-axis caption. In some cases, some observationds are filtered to
show a clearer picture, this is indicated with filter.
Standard Error
Rho (only pairs on same branch)
Effect of Parameter Estimate
Hetereogeneity
The data suggests a step-like relationship such that only values that
are at or near zero show a considerable probability of non-zero absolute
deviations. To look at this further, we can see how probable it is to
observe values near zero. The following table shows that at least 80% of
observations have a log1p value that is very near to
zero.
## # A tibble: 8 x 3
## cond_x less_than_00001 less_than_01
## <fct> <dbl> <dbl>
## 1 Comp MLE 0.791 0.886
## 2 Comp Bayes 0.791 0.886
## 3 No asy 0.791 0.886
## 4 No PB 0.791 0.885
## 5 No NPB 0.807 0.902
## 6 No Bayes 0.781 0.880
## 7 Beta PP 0.789 0.884
## 8 Trait_u PP 0.852 0.894If we look at the conditionmal disttirbution of absolute deviation whether or not it is very near to zero, we can see that there is some evidence for the step-like relationship, but the pattern is not overwhelming.
Rho
Fungibility
Fungibility (only pairs on same branch)
Model Fit
Proportion of Individual Misfit
Relative Parameter Information
The reason both plots look pretty much the same is that both relative
parameter information variables are highly correlated, \(r \approx 1\). We therefore focus on one of
the two below (rel_par_weight_y).
Using a logarithm:
And removing all with a relative weight of roughly 1:
Relative Parameter N
We can also consider the relative N. As tehse are again highly
correlated (\(r \approx 1\)), we use
y again exclusivly:
Here it makes sense to trim the x-axis a bit:
Effects of Categroical Covariates
The effect of covariate is shown in two ways. The table below all plots gives the RMSEs for the model given the covariate across all comparison methods. In case the number of levels is not too large, a plot of the difference in absolute deviation conditional on the factor levels is shown. Some factor levels may be removed for plotting (i.e., those levels for which the proportion of observations is less than 0.04). In this case, the number of removed levels is also given.
Meta Data
## # A tibble: 4 x 10
## covariate nlevels `Comp MLE` `No asy` `No PB` `No NPB` `No Bayes` `Comp Bayes` `Trait_u PP` `Beta PP`
## <chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 model 8 0.0599 0.0891 0.0580 0.0436 0.0953 0.0912 0.0624 0.0554
## 2 model2 12 0.0907 0.112 0.0767 0.0666 0.152 0.0944 0.137 0.0917
## 3 parameter 49 0.362 0.462 0.385 0.344 0.407 0.134 0.421 0.318
## 4 dataset 147 0.191 0.222 0.194 0.208 0.397 0.729 0.292 0.286Given that there are more than eight levels of model2,
we also look at a table of the mean absolute deviations:
## # A tibble: 12 x 10
## model2 prop `Comp MLE` `Comp Bayes` `No asy` `No PB` `No NPB` `No Bayes` `Beta PP` `Trait_u PP`
## <fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2htsm_4 0.301 0.0199 0.0178 0.0284 0.0369 0.0352 0.0471 0.0180 0.00671
## 2 2htsm_5d 0.0890 0.0420 0.0303 0.0438 0.0661 0.0561 0.115 0.0313 0.0158
## 3 2htsm_6e 0.0226 0.0415 0.0357 0.0418 0.0571 0.0499 0.105 0.0326 0.0197
## 4 c2ht6 0.0271 0.0331 0.0304 0.0433 0.0431 0.0431 0.0406 0.0394 0.0230
## 5 c2ht8 0.00565 0.0263 0.0257 0.0242 0.0271 0.0268 NA 0.00591 0.0399
## 6 pd_s 0.0316 0.0114 0.0117 0.0180 0.0208 0.0207 0.0419 0.0306 0.00351
## 7 pd_e 0.00753 0.0462 0.0259 0.0942 0.0867 0.0882 0.119 0.0959 0.0721
## 8 pm 0.105 0.0369 0.0910 0.0351 0.0412 0.0423 0.117 0.0405 0.0149
## 9 hb 0.0524 0.0518 0.0303 0.0823 0.0864 0.0596 0.0827 0.0312 0.00640
## 10 rm 0.149 0.0142 0.0144 0.0165 0.0164 0.0164 0.0233 0.0195 0.00249
## 11 real 0.104 0.0173 0.0138 0.0244 0.0276 0.0249 0.0453 0.0133 0.00608
## 12 quad 0.105 0.0336 0.0276 0.0455 0.0624 0.0491 0.0712 0.0293 0.0190We can also look at a table of the mean absolute deviations as a function of the parameter:
## # A tibble: 49 x 10
## parameter prop `Comp MLE` `Comp Bayes` `No asy` `No PB` `No NPB` `No Bayes` `Beta PP` `Trait_u PP`
## <fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2htsm_4:b 0.0752 0.0187 0.0184 0.0264 0.0423 0.0421 0.0393 0.0168 0.00570
## 2 2htsm_4:d 0.0752 0.0332 0.0274 0.0514 0.0640 0.0560 0.0976 0.0293 0.0134
## 3 2htsm_4:D 0.0752 0.0101 0.00858 0.0170 0.0160 0.0160 0.0260 0.0109 0.00239
## 4 2htsm_4:g 0.0752 0.0175 0.0168 0.0190 0.0251 0.0266 0.0255 0.0149 0.00536
## 5 2htsm_5d:b 0.0178 0.0327 0.0337 0.0218 0.0247 0.0240 0.0699 0.0294 0.0113
## 6 2htsm_5d:D 0.0178 0.0104 0.0100 0.0152 0.0151 0.0151 0.0301 0.0109 0.00300
## 7 2htsm_5d:d_1 0.0178 0.0674 0.0447 0.0813 0.122 0.0983 0.217 0.0494 0.0247
## 8 2htsm_5d:d_2 0.0178 0.0733 0.0447 0.0729 0.135 0.111 0.203 0.0522 0.0299
## 9 2htsm_5d:g 0.0178 0.0261 0.0185 0.0275 0.0339 0.0318 0.0533 0.0148 0.0103
## 10 2htsm_6e:b 0.00377 0.0481 0.0486 0.0300 0.0155 0.0163 0.0905 0.0449 0.0142
## 11 2htsm_6e:d_1 0.00377 0.0469 0.0340 0.0829 0.151 0.129 0.214 0.0443 0.0253
## 12 2htsm_6e:D_1 0.00377 0.0197 0.0204 0.0223 0.0101 0.0116 0.0673 0.0194 0.00269
## 13 2htsm_6e:d_2 0.00377 0.0832 0.0639 0.0714 0.128 0.102 0.117 0.0474 0.0510
## 14 2htsm_6e:D_2 0.00377 0.0193 0.0197 0.0241 0.0135 0.0162 0.0721 0.0217 0.00512
## 15 2htsm_6e:g 0.00377 0.0321 0.0275 0.0201 0.0244 0.0241 0.0680 0.0181 0.0198
## 16 c2ht6:Dn 0.00904 0.0580 0.0520 0.0671 0.0702 0.0702 0.0683 0.0711 0.0414
## 17 c2ht6:Do 0.00904 0.0171 0.0169 0.0282 0.0271 0.0271 0.0265 0.0159 0.0100
## 18 c2ht6:g 0.00904 0.0244 0.0222 0.0345 0.0319 0.0319 0.0269 0.0312 0.0175
## 19 c2ht8:Dn 0.00188 0.0507 0.0492 0.0499 0.0538 0.0528 NA 0.00737 0.0857
## 20 c2ht8:Do 0.00188 0.0117 0.0117 0.00825 0.0135 0.0135 NA 0.00627 0.00814
## 21 c2ht8:g 0.00188 0.0166 0.0162 0.0144 0.0141 0.0141 NA 0.00411 0.0259
## 22 hb:b 0.0131 0.0968 0.0559 0.208 0.219 0.158 0.115 0.0462 0.0160
## 23 hb:c 0.0131 0.0906 0.0472 0.0871 0.0916 0.0366 0.158 0.0573 0.000630
## 24 hb:rc 0.0132 0.00813 0.00659 0.0168 0.0177 0.0240 0.0304 0.00808 0.00340
## 25 hb:re 0.0131 0.0117 0.0115 0.0169 0.0204 0.0199 0.0275 0.0130 0.00559
## 26 pd_e:A_alt 0.00452 0.0404 0.0310 0.0665 0.0655 0.0655 0.104 0.0650 0.0583
## 27 pd_e:C_Exclusion 0.00151 0.0191 0.0111 0.0714 0.0701 0.0701 0.0902 0.0873 0.0291
## 28 pd_e:C_Inclusion 0.00151 0.0908 0.0255 0.200 0.167 0.174 0.192 0.197 0.157
## 29 pd_s:A 0.0158 0.00995 0.0102 0.00724 0.00879 0.00867 0.0180 0.0124 0.00499
## 30 pd_s:C 0.0158 0.0129 0.0132 0.0288 0.0327 0.0327 0.0658 0.0488 0.00203
## 31 pm:C1 0.0264 0.0169 0.0798 0.0271 0.0256 0.0250 0.0908 0.0248 0.00858
## 32 pm:C2 0.0264 0.0150 0.0885 0.0141 0.0163 0.0163 0.0949 0.0177 0.00740
## 33 pm:M 0.0264 0.0607 0.129 0.0409 0.0578 0.0615 0.183 0.0454 0.0284
## 34 pm:P 0.0264 0.0550 0.0663 0.0581 0.0651 0.0663 0.0979 0.0740 0.0153
## 35 quad:ACbb1 0.0210 0.0172 0.0173 0.0364 0.0385 0.0385 0.0726 0.0203 0.00281
## 36 quad:ACwg1 0.0210 0.0178 0.0182 0.0356 0.0378 0.0378 0.0708 0.0197 0.00247
## 37 quad:D1 0.0210 0.0207 0.0214 0.0126 0.0124 0.0124 0.0256 0.0256 0.00186
## 38 quad:G1 0.0210 0.0138 0.0140 0.0218 0.0229 0.0247 0.0156 0.0105 0.0105
## 39 quad:OB1 0.0210 0.0987 0.0671 0.121 0.200 0.132 0.171 0.0706 0.0772
## 40 real:A1 0.0147 0.0106 0.0103 0.0158 0.0293 0.0183 0.0314 0.00993 0.00767
## 41 real:A2 0.0147 0.00867 0.00653 0.0115 0.0274 0.0153 0.0252 0.00711 0.00580
## 42 real:L1 0.0147 0.0161 0.0136 0.0107 0.00896 0.0101 0.0372 0.00948 0.00340
## 43 real:L2 0.0147 0.0163 0.0131 0.0133 0.0110 0.0115 0.0369 0.00985 0.00304
## 44 real:L3 0.0152 0.00791 0.00579 0.0162 0.0170 0.0162 0.0540 0.0120 0.00339
## 45 real:L4 0.0152 0.00734 0.00486 0.0140 0.0134 0.0138 0.0523 0.00983 0.00278
## 46 real:Re 0.0147 0.0544 0.0423 0.0890 0.0932 0.0899 0.0802 0.0351 0.0165
## 47 rm:a 0.0497 0.00448 0.00458 0.00491 0.00492 0.00492 0.00936 0.00616 0.000742
## 48 rm:b 0.0497 0.00650 0.00648 0.00628 0.00618 0.00612 0.00811 0.00350 0.00166
## 49 rm:r 0.0497 0.0317 0.0321 0.0383 0.0382 0.0382 0.0526 0.0487 0.00508Or at the standard deviation of the absolute deviations as a function of the parameter:
## # A tibble: 49 x 10
## parameter prop `Comp MLE` `Comp Bayes` `No asy` `No PB` `No NPB` `No Bayes` `Beta PP` `Trait_u PP`
## <fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2htsm_4:b 0.0752 0.0227 0.0221 0.0313 0.0595 0.0581 0.0426 0.0190 0.00764
## 2 2htsm_4:d 0.0752 0.0354 0.0260 0.0486 0.0883 0.0595 0.0831 0.0249 0.0178
## 3 2htsm_4:D 0.0752 0.0109 0.00983 0.0162 0.0150 0.0150 0.0214 0.0114 0.00358
## 4 2htsm_4:g 0.0752 0.0288 0.0283 0.0185 0.0323 0.0358 0.0294 0.0229 0.00780
## 5 2htsm_5d:b 0.0178 0.0217 0.0216 0.0136 0.0179 0.0164 0.0352 0.0187 0.0111
## 6 2htsm_5d:D 0.0178 0.00726 0.00781 0.00632 0.00639 0.00639 0.0216 0.00804 0.00312
## 7 2htsm_5d:d_1 0.0178 0.0453 0.0278 0.0554 0.0863 0.0622 0.111 0.0318 0.0227
## 8 2htsm_5d:d_2 0.0178 0.0505 0.0402 0.0538 0.0871 0.0588 0.113 0.0443 0.0268
## 9 2htsm_5d:g 0.0178 0.0240 0.0176 0.0175 0.0185 0.0207 0.0316 0.0104 0.0101
## 10 2htsm_6e:b 0.00377 0.0282 0.0286 0.0245 0.0152 0.0168 0.0548 0.0280 0.0121
## 11 2htsm_6e:d_1 0.00377 0.0577 0.0424 0.0711 0.119 0.107 0.126 0.0295 0.0309
## 12 2htsm_6e:D_1 0.00377 0.00815 0.00828 0.00433 0.00446 0.00346 0.0175 0.00614 0.00224
## 13 2htsm_6e:d_2 0.00377 0.0468 0.0466 0.0317 0.0580 0.0421 0.145 0.0388 0.0214
## 14 2htsm_6e:D_2 0.00377 0.0117 0.0118 0.0188 0.0130 0.0121 0.0332 0.0107 0.00205
## 15 2htsm_6e:g 0.00377 0.0282 0.0179 0.0156 0.0327 0.0328 0.0217 0.0128 0.0183
## 16 c2ht6:Dn 0.00904 0.0332 0.0280 0.0521 0.0520 0.0520 0.0494 0.118 0.0614
## 17 c2ht6:Do 0.00904 0.00981 0.00999 0.0141 0.0125 0.0125 0.0146 0.0192 0.0158
## 18 c2ht6:g 0.00904 0.0141 0.0133 0.0214 0.0172 0.0172 0.0203 0.0570 0.0345
## 19 c2ht8:Dn 0.00188 0.0230 0.0224 0.00349 0.00814 0.00691 NA 0.00941 0.0644
## 20 c2ht8:Do 0.00188 0.00966 0.00967 0.00797 0.0182 0.0182 NA 0.00546 0.00860
## 21 c2ht8:g 0.00188 0.0114 0.0112 0.00127 0.00259 0.00259 NA 0.00528 0.0225
## 22 hb:b 0.0131 0.0567 0.0400 0.110 0.132 0.108 0.0740 0.0398 0.0113
## 23 hb:c 0.0131 0.0954 0.0451 0.0954 0.0948 0.0443 0.0789 0.0621 0.000634
## 24 hb:rc 0.0132 0.00746 0.00747 0.0323 0.0330 0.0458 0.0286 0.00858 0.00418
## 25 hb:re 0.0131 0.0115 0.0118 0.0252 0.0279 0.0324 0.0242 0.0123 0.00656
## 26 pd_e:A_alt 0.00452 0.0205 0.0251 0.0471 0.0501 0.0501 0.0794 0.0142 0.0207
## 27 pd_e:C_Exclusion 0.00151 0.0175 0.00308 0.0533 0.0551 0.0551 0.0548 0.0706 0.0120
## 28 pd_e:C_Inclusion 0.00151 0.0490 0.0261 0.0666 0.0199 0.0303 0.214 0.0164 0.0403
## 29 pd_s:A 0.0158 0.00887 0.00909 0.00790 0.00875 0.00881 0.0219 0.0143 0.00468
## 30 pd_s:C 0.0158 0.0110 0.0112 0.0212 0.0219 0.0219 0.0437 0.0729 0.00201
## 31 pm:C1 0.0264 0.0167 0.206 0.0289 0.0291 0.0293 0.142 0.0235 0.0157
## 32 pm:C2 0.0264 0.0138 0.242 0.0172 0.0161 0.0161 0.150 0.0135 0.0206
## 33 pm:M 0.0264 0.0791 0.246 0.0503 0.0456 0.0492 0.139 0.0533 0.0458
## 34 pm:P 0.0264 0.0452 0.0733 0.0516 0.0523 0.0538 0.0738 0.0588 0.0138
## 35 quad:ACbb1 0.0210 0.0122 0.0122 0.0156 0.0148 0.0148 0.0189 0.0112 0.00269
## 36 quad:ACwg1 0.0210 0.0106 0.0103 0.0130 0.0126 0.0126 0.0180 0.00962 0.00228
## 37 quad:D1 0.0210 0.0144 0.0143 0.0117 0.0118 0.0118 0.0142 0.0161 0.00205
## 38 quad:G1 0.0210 0.00913 0.00906 0.0165 0.0172 0.0142 0.0120 0.00702 0.00685
## 39 quad:OB1 0.0210 0.0880 0.0554 0.0797 0.176 0.112 0.107 0.0601 0.0623
## 40 real:A1 0.0147 0.0118 0.0119 0.0209 0.0700 0.0352 0.0421 0.0102 0.0116
## 41 real:A2 0.0147 0.00560 0.00539 0.00698 0.0602 0.0240 0.0167 0.00521 0.00662
## 42 real:L1 0.0147 0.0145 0.0119 0.00798 0.00754 0.00725 0.0174 0.00622 0.00238
## 43 real:L2 0.0147 0.0129 0.0125 0.0127 0.00954 0.00905 0.0187 0.00792 0.00269
## 44 real:L3 0.0152 0.00645 0.00465 0.00695 0.00672 0.00695 0.0127 0.00635 0.00296
## 45 real:L4 0.0152 0.00596 0.00399 0.00737 0.00704 0.00702 0.0123 0.00555 0.00209
## 46 real:Re 0.0147 0.0374 0.0305 0.0844 0.170 0.132 0.0861 0.0298 0.0130
## 47 rm:a 0.0497 0.00564 0.00573 0.00607 0.00606 0.00606 0.00860 0.00718 0.00147
## 48 rm:b 0.0497 0.00732 0.00726 0.00623 0.00666 0.00650 0.00502 0.00232 0.00289
## 49 rm:r 0.0497 0.0227 0.0225 0.0339 0.0342 0.0342 0.0365 0.0384 0.0220Categorical Covariates
## # A tibble: 2 x 10
## covariate nlevels `Comp MLE` `No asy` `No PB` `No NPB` `No Bayes` `Comp Bayes` `Trait_u PP` `Beta PP`
## <chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 population 5 0.0273 0.0477 0.0192 0.0132 0.00459 0.000836 0.00622 0.0168
## 2 sci_goal 2 0.000177 0.00000983 0.000419 0.0000276 0.00384 0.00000328 0.0111 0.00139